PiBeat

Introduction

PiBeat is a reaction time game played using a RaspberryPi with a piTFT connected to an accelerometer and a 32x16 LED matrix. On boot, the game screen automatically starts at the home screen using cron. The user uses the piTFT to select which team they want to be on, which will contribute to the leaderboard score. In order to start, stop, or resume game play, the user will use the corresponding buttons on the piTFT. The game is then displayed on the LED matrix where instructions flow down the board, indicating the desired orientation of the accelerometer. The player times their actions to orient the accelerometer as directed when the instructions reach the bottom of the screen, and they are awarded points based on how accurately their actions are timed. The game ends when the player reaches a threshold number of missed instructions. The system is driven by a main game loop, which periodically polls the accelerometer and produces output signals for the LED matrix. The game loop also generates random instructions and instruction positions, which adds an unpredictable component to the game.

Hardware Design and Testing

The hardware of our project consisted of a Raspberry Pi 4, a touch screen PiTFT, the Adafruit MMA8451 accelerometer, and a 32x16 LED Matrix. To turn on the Raspberry Pi, we connected a 3v3 power source to the component and inserted a 16 GB SD card. To power the PiTFT, we connected the PiTFT to the Raspberry Pi 4 using a 40-pin cable header such that the PiTFT can read information on the relevant GPIO pins. All of the pins on the Adafruit MMA8451 accelerometer were first soldered to maintain reliable electrical contact when utilizing the accelerometer. We then connected the VIn, GND, SCL, and SDA input on the accelerometer to Pin 1, Pin 6, Pin 2, and Pin 3 respectively on the Raspberry Pi 4 as seen in Table 1. With these connections, the accelerometer was able to detect which of the 8 standard adafruit orientations the accelerometer was currently in. To power the 32x16 LED Matrix, we connected 1 5V power source to the matrix and connected 16 pins from the Raspberry Pi 4 to the LED matrix such that the LED Matrix can read information on the relevant pins as shown in Table 2. An overall diagram of the system and communication between the 4 main different components can be seen in Figure 1.

Accelerometer Installation and Testing

The first external hardware component we installed was the Adafruit MMA8451 Accelerometer, and to test our ability to interface with it, we had to install various system dependencies before running any testing scripts. Since the Adafruit MMA8451 module uses CircuitPython, we installed Blinka, which is a compatibility package that allows code written using CircuitPython to be run on Linux systems on which Python is installed [11]. We also installed the Linux package dependencies i2c-tools, libgpiod-dev, and python3-libgpiod. The i2c-tools package allows a method for interfacing with I2C devices, and it was critical for testing the correct wiring and setup of the accelerometer. The libgpiod-dev and python3-libgpiod packages provide for interaction between the Linux system and GPIO pins, as well as bindings that facilitate GPIO operations in Python. After installing Blinka, we tested our installation by running ls /dev/i2c*, which allowed us to observe the output /dev/i2c-1, indicating that I2C was properly enabled. We also ran the script mma_setup/blinka_test.py provided by Adafruit, and we verified we were able to create the I2C device corresponding to the accelerometer through Python [11].

During the setup of Blinka, one issue we encountered was that running i2cdetect -y 1, which is a command that scans the I2C bus and indicates the address of any detected devices, did not consistently detect the presence of the accelerometer. We later realized this was due to an unreliable connection between the accelerometer and the header pins, and the issue was fixed after we resoldered a joint that was not properly connected.

Next, we ran mma_setup/mma_simple.py and mma_setup/simpletest.py, which were Adafruit test scripts used to verify we could accurately read orientation values from the accelerometer [11]. In particular, we confirmed that we are able to read each of the eight output values of the accelerometer: Portrait Up Front (PUF), Portrait Up Back (PUB), Portrait Down Front (PDF), Portrait Down Back (PDB), Landscape Right Front (LRF), Landscape Right Back (LRB), Landscape Left Front (LLF), and Landscape Left Back (LLB). We faced a number of significant issues in running this; in particular, we inconsistently received I/O errors while trying to read from the accelerometer. This error was difficult to resolve since the error occurred spuriously, but we ultimately realized the communication rate expected by the Raspberry Pi on the I2C bus was much higher than what was provided by the accelerometer. We resolved this by setting dtparam=i2c_arm_baudrate=10000 in /boot/config.txt. However, this change only seemed to work on the patched kernel from Lab 4 and refused to operate with the Lab 3 kernel, so we decided to build PiBeat on top of the RT patch kernel.

LED Matrix Installation and Testing

The installation of the LED matrix was unexpectedly one of the most challenging steps of our project. After initially wiring the LED matrix as described previously, we installed the rpi-rgb-led-matrix library using the Adafruit installer script [4]. According to this script, the various sample programs provided with the installation were expected to run smoothly after configuring a few parameters, such as matrix size and GPIO slowdown. However, even after configuring these and rigorously checking all the components of our installation, we were unable to generate any signals on the LED matrix. We believed this was due to a hardware issue or incompatibility with the kernel, as we were building off the RT patch kernel.

To investigate this, we started by reverting to a clean backup of the Lab 3 kernel, which did not have the RT patch applied. We ran the same installation script on the previous kernel, but we found the LED matrix still had no response. We then spent significant amounts of time replacing each individual component of our system, ranging from jumper wires to DC power adapters to Raspberry Pi ribbon cables, until we ultimately found our original LED matrix was simply broken. After obtaining a new LED matrix, we were able to run the sample scripts provided with the rpi-rgb-led-matrix installation [7]. To more thoroughly test and explore the library, we modified the scripts slightly to perform actions more akin to the controls of PiBeat, such as by displaying custom matrices and setting individual pixels of the LED matrix.

Software Design and Testing

We followed an incremental development process in integrating each hardware component of our project, as we aimed to ensure each component worked individually before attempting to combine separate parts. In doing so, we were able to identify many installation issues in isolation and understand the interfaces for communication between each component before developing the game logic.

Overall System Design

The main control for PiBeat is pibeat_led.py, which contains both game logic and interfacing with the external hardware devices. This script also creates the displays to be shown on the PiTFT, and it maintains a persistent version of the leaderboard by interacting with the file system. The script maintains a global variable menu_level which is generally used to indicate which stage the user is currently at within the menu system. When menu_level is 3, this indicates the user is actively playing the game, which enables a game loop described in the Game Logic section. To effectively coordinate our LED matrix with the game mechanics, we spawn a thread at the beginning of pibeat_led.py whose target is the run method of the LED matrix, as described in the LED Matrix Signal Generation section. The main game loop communicates with the LED matrix via a designated set of global variables.

Game Logic

Once the user begins a game, the main game loop begins. Each iteration of the loop starts by sampling the current orientation of the accelerometer, which becomes the value considered as the user’s input for the current round. Each round of the game maintains a variable game_time which is independent of the overall script running time; instead, game_time is a critical component of the game logic that dictates the current time step of the game. A new instruction is generated every five time steps, and these instructions are chosen randomly from the orientation enums provided by the Adafruit MMA8451 interface. We exclude PUF, which we consider the neutral position, and we add a None option that allows for gaps in the instruction stream. Newly generated instructions are appended to the list of active instructions, stored in the global variable cur_insns. This is maintained as a list of (instruction, generated_time) pairs, where we maintain the generated time as convenience for calculating scores.

The next step is to evaluate the next upcoming instruction and compare it with the current orientation provided by the user. We check if the instruction is expired, which is defined as when game_time > generated_time+2, and we remove it if it is. Otherwise, if some non-neutral orientation was detected from the user, we evaluate the current instruction in comparison with the provided orientation. If the orientation and instruction match, we evaluate the time at which the instruction was hit relative to its “target time”, defined through a helper function get_target_time, to determine the number of points to award to the player. The current game state is then evaluated to determine if the game should be ended by checking the number of instructions missed, and the variable game_time is then incremented. Once the game is over, menu_level transitions to 6, which exits the game loop and resets all local game variables such as game_time and cur_insns.

Since we designed the game logic loop very clearly before implementing it, we were able to implement the basic game with very little obstacles. We also developed the program incrementally, starting with generating single instructions and comparing pairs of instructions and orientations, and gradually moving to a more complex system.

LED Matrix Signal Generation

Interfacing with the LED matrix was one of the most challenging parts of our project, as we initially intended to build our interface off of the SampleBase object provided in the rpi-rgb-led-matrix library but soon discovered that the logical structure of SampleBase was not compatible with our iterative game loop [7]. This led us to create our own matrix class, RunMatrix, which initializes and updates an RGBMatrix object by setting each individual pixel of the 16x32 matrix.

During each time step of the game loop, RunMatrix also takes a single step. To construct the matrix to be displayed on the LED matrix, RunMatrix constructs a 16x32 matrix by piecing together two helper functions defined in runmatrix_helpers.py, get_insn_leds and get_bottom_leds.

The first helper function, get_insn_leds, is responsible for transforming game state into a 16x25 matrix of instructions. In particular, get_insn_leds takes as arguments cur_insns and game_time, transforms the list of instructions into 5x16 matrices containing arrows, and determines the correct slice of these matrices to display based on game_time. In order to implement random offsets for our arrows, which are needed to ensure the arrows do not appear at predictable locations on the LED matrix, we hash the generated_time of each instruction and use the modulus operator to ensure we can deterministically map each instruction to a valid randomized offset.

The second helper function, get_bottom_leds, is responsible for displaying the feedback about any actions to the user. In particular, the main game loop sets a variable feedback based on the timing of a user’s action, and each possible value of feedback maps to a specific color gradient to be displayed to the user.

PiTFT Main UI

The PiTFT UI is organized into six menu levels, indicated by the global variable menu_level. To facilitate the development of each menu, we defined menu_display_dict and menu_event_dict, where each maps each possible value of menu_level to a unique handler for generating the current display and handling touch events, respectively. Touch events are detected using Pygame, which provides a queue of touch events and their corresponding touch locations. We iterate through this queue in each iteration of the main game loop to ensure we remain responsive to interactions with the PiTFT touchscreen.

The starting menu level displays a basic welcome screen, as shown below.

The second level of the menu allows the player to select their team. Each of the four teams, ice, rain, ocean, and sky represent blue items within nature as the color blue is often associated with calmness, so the game UI concentrates on various forms of blue to emphasize the idea that PiBeat is a relaxing game that people play for fun. In order to increase the team spirit within each of the four teams, we utilized a graphical font and 3D text effects website [14] to personalize each team based on the traits of their name as seen below.

During the game, the UI indicates the player’s current score and remaining misses, which allows them to track their progress throughout the game. Once a game is finished, the UI also displays the user’s overall score, global ranking, and any new high scores earned.

The primary challenge with creating the UI was creating an effective abstraction for our menu levels and ensuring they did not conflict with one another. We ultimately decided on the map structure because it provided efficient lookups for handlers and allowed us to develop each menu level independently, and it also facilitates modification and expansion of the menu system.

Leaderboard

The leaderboard system of PiBeat is implemented by storing two lists in a persistent file. The first list contains all scores earned by all players, maintained in ascending sorted order. This allows us to easily identify the sorted position of any newly generated scores and efficiently determine the global ranking and insertion index of new scores. The second list contains the high scores of each of the four teams. These lists are read into memory during the execution of PiBeat, and they are written back to persistent storage in every termination of an execution of the game. This ensures that scores are maintained across different runs of the program.

cron Scheduling

To have our game start automatically each time the Raspberry Pi is booted, we created a script pibeat.sh that calls pibeat_led.py. We initially had issues with running this script because we forgot to modify the permissions such that the script is executable, but once we corrected this, we were able to manually run pibeat.sh. To have it run using cron, we modified crontab and called the script using the tag @reboot.

Results

By the end of the project timeline, the game performed as we had planned and all of the core goals that we wanted to accomplish were met as outlined in the description. We were able to wire the accelerometer, LED display board, and piTFT to the Raspberry Pi 4 by utilizing a breadboard such that changes in display were synchronized across the piTFT and LED display board and changes sensed by the accelerometer were consistently read. Despite our initial issues with setting up the hardware to interact with the Raspberry Pi 4 nicely, we were able to finish on time and incorporate many features into the game through the software.

The game successfully incorporates an aesthetically pleasing user interface that is inspired by the beauty and tranquility of nature by utilizing a blue based color palette and promotes collective ethos with the customization of the game based on which team the player chooses. Connections between the piTFT and the LED display board are highly coupled and transitions from one stage of the game to the next on either end are shown on both display types at a highly synchronized rate, maintaining the visually appealing manner of the game.

After choosing a team, a player has officially entered the game and is able to look at the leaderboard, which shows the highest score ever gained for each team on the piTFT and displays that team on the LED display board in order to motivate the player to improve their team’s performance. The player can also choose to start a new game and follow the randomly generated set of instructions by trying to move the controller to the specified orientation on beat when the instruction falls to the end of the screen. The game maintains different point values based on how well the player was on beat, by rewarding the player with 10 points for being perfectly on time with when the middle of the instruction passes the middle, 5 points for being slightly off time from the middle, and 3 points for aligning with the edges of the instruction. At the same time, the bottom of the LED display board also shows a gradient representing the awarded amount to quickly allow the player to evaluate their performance and adjust as needed. Once the player completely misses a set number of misses, the game ends and the player is automatically notified of how well their current score is ranked among all historical scores and whether or not they were able to create a new high for their team.

The game is automatically displayed on the piTFT at startup due to the cron job which allows for players to play the entire game with just the listed components without the need to login to the Raspberry Pi or use a mouse or keyboard to type into the command line. The user can also return to any of the initial screens using the back button and quit button to maximize a satisfactory user experience.

Conclusion

PiBeat was able to create a fully autonomous and functional reaction time game that utilizes two different types of displays to bring an aesthetically pleasing UI to the player and a leveled reward system to update the score the player gains based on how well their reaction is to the set of instructions by leveraging multithreading.

One of the main issues that occurred during the entire creation of the game was setting up hardware to function properly. Since most of the software is game logic that controls outputs of the display components using inputs from the accelerometer and other internal signals, it was critical that all of the hardware was set up correctly before most of the software could be developed. However, due to differences in documentation of different pieces of the hardware and issues with some of the components, themselves, we spent a lot of time debugging the issue with the hardware components by slowly switching out each connecting piece including the jumper wires, kernel type, rewiring jumper wires, and the actual physical components like the Raspberry Pi, accelerometer, and display board.

On the software side of the project, we realized that it was impossible to build solely off samplebase.py within the rpi-rgb-led-matrix [7] repository, since it was customized to display a determined set onto the LED display, so a section of the project was dedicated to creating a custom class that combines the setup within samplebase.py and an iterative update to the LED display screen that was able to take inputs while running to change the actual matrix displayed onto the screen.

Overall, after setting up the entire hardware and customizing parts of the software to optimize timing throughout the different hardware components, we were able to create a synchronized system across all of the components that allowed for comfortable game play.

Future Work

The current setup for the hardware of PiBeat allows the entire game to run smoothly and work, however it is not optimized for a player to use, since all of the wires are exposed and risk being potentially pulled apart. One main improvement to this setup would be to 3D print a casing for the Raspberry Pi, piTFT, LED display board, and breadboard setup such that the piTFT screen is right above the LED display board screen and the controller that contains the accelerometer is controlled outside of the casing.

The PiBeat game is a very steep learning curve, since it takes some time to understand how each instruction shown on the LED display screen correlates with the directions shown on the controller. To make this learning process easier and more understandable, it would be nice to have an interactive instruction demo that would slowly show each instruction to the player and have the player perform that instruction repeatedly until the player is right. We could also incorporate a quick help screen that would visually go over all of the different directions and the meaning of the colors of the arrows on the piTFT.

Since the PiBeat game is very difficult, there is currently only one difficulty mode for the game that incorporates all of the different instructions at a slow speed. We wish to make the game more playable such that there are easier levels that have fewer instructions at a slow speed and more difficult levels that are at a faster speed. This would be easiest to implement by adding a screen on the piTFT that would allow the player to set the game mode by choosing the range of instruction types they want to see (for example, only forward facing orientation instructions) and the speed at which the instructions update.

Finally, we both believe it would be very interesting to incorporate a computer vision game solver to PiBeat by utilizing a robot arm to maneuver the controller with a camera to see the instructions that are being shown on the LED display screen and moving the controller to the right position based on the directions it sees.

Parts List

• Raspberry Pi 4, 2G Ram - $35.00
• Capacitive piTFT - $44.95
• Raspberry Pi extension cable - $2.95
• Raspberry Pi header - $1.50
• Raspberry Pi case - $4.99
• Raspberry Pi power supply - $8.00
• 16G SD card - $8.00
• Adafruit MMA8451 Accelerometer - $7.95
• Adafruit 16x32 RGB LED Matrix Display - 6mm Pitch - $24.95
• 5V 2A DC Power Adapter - $9.00
• Female DC Power Adapter - $2.00
• 40 Breadboard Jumper Wires - $5.00
• Total $154.29

ECE 5725 Required Components (Raspberry Pi 4, Capacitive piTFT, Raspberry Pi extras, 16G SD card) - $105.39

Total: $48.90

References

[1] Adafruit. Adafruit MMA8451 Accelerometer Breakout. Adafruit Learn. Accessed: Apr. 30, 2024. [Online]. Available: https://learn.adafruit.com/adafruit-mma8451-accelerometer-breakout

[1] Adafruit. Adafruit MMA8451 Accelerometer Breakout. Adafruit Learn. Accessed: Apr. 30, 2024. [Online]. Available: https://learn.adafruit.com/adafruit-mma8451-accelerometer-breakout

[2] Adafruit. Adafruit RGB Matrix Bonnet for Raspberry Pi. Adafruit Learn. Accessed: May 02, 2024. [Online]. Available: https://learn.adafruit.com/adafruit-rgb-matrix-bonnet-for-raspberry-pi

[3] Adafruit. P420 Indoor-P6-8S-16x32. IS-P6-8S-SMD3528 datasheet. Aug. 2020. https://cdn-shop.adafruit.com/product-files/420/P420_Indoor-P6-8S-16x32-SMD3528.pdf

[4] Adafruit, Raspberry Pi Installer Scripts. Github. Accessed: May 02, 2024. [Online]. Available: https://github.com/adafruit/Raspberry-Pi-Installer-Scripts/blob/main/rgb-matrix.sh

[5] Freepik.com. Back Icon. Freepik. Accessed: May 04, 2024. [Online]. Available: https://www.freepik.com/icon/back_2099190

[6] H. Pranowo. Trophy With Star. Vecteezy. Accessed: May 04, 2024. [Online]. Available: https://www.vecteezy.com/vector-art/17447796-best-collection-icon-symbol-championship-or-competition-trophy-with-star

[7] H. Zeller, rpi-rgb-led-matrix. Github. Accessed: May 12, 2024. [Online]. Available: https://github.com/hzeller/rpi-rgb-led-matrix

[8] J. Skovira. ECE 5725: Lab 1 Manual, version 7 (2024). Accessed: May 02, 2024. [Online]

[9] J. Skovira. ECE 5725: Lab 4 Manual, version 5 (2024). Accessed: May 02, 2024. [Online]

[10] J. Skovira. SD Card Backup, version 2 (2024). Accessed: May 12, 2024. [Online]

[11] M. LeBlanc-Williams. Installing Blinka on Raspberry Pi. Adafruit Learn. Accessed: Apr. 30, 2024. [Online]. Available: https://learn.adafruit.com/circuitpython-on-raspberrypi-linux/installing-circuitpython-on-raspberry-pi

[12] NXP Semiconductors. MMA8451Q, 3-axis, 14-bit/8-bit digital accelerometer. MMA8451Q datasheet. Feb. 2017. https://www.nxp.com/docs/en/data-sheet/MMA8451Q.pdf

[13] T. Bisk. Live Scoreboard on Raspberry Pi. ECE 5725 Fall 2023 Projects. Accessed: May. 12, 2024. [Online]. Available: https://courses.ece.cornell.edu/ece5990/ECE5725_Fall2023_Projects/2%20Wednesday%20December%206/1%20Live%20Scoreboard/M_tjb/index.html

[14] Text Studio. Font Generator & 3D Text Effects. TextStudio. Accessed: May 04, 2024. [Online]. Available: https://www.textstudio.com/